Control device, camera system and program

ABSTRACT

A control device of present invention includes a display control portion that performs control such that a whole image display region, in which a whole image captured by a camera is displayed, and an enlarged image display region, in which a region corresponding to a position specified on the whole image is enlarged and an enlarged image is displayed, are displayed, and a control portion that performs control such that the region corresponding to the position specified on the whole image is changed. The region includes at least one preset region, and the display control portion displays, on the whole image, a section that corresponds to the preset region.

CROSS-REFERENCE TO PREVIOUS APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/932,945 filed on Mar. 10, 2011, which claims priority from JapanesePatent Application No. JP2010-073232 filed in the Japanese Patent Officeon Mar. 26, 2010, the entire contents of which are incorporated hereinby reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a control device, a camera system and aprogram.

2. Description of the Related Art

A camera control system is known in which a range that can be capturedby a camera is displayed as a whole image and an imaging direction ofthe camera is controlled by an operation on the whole image. This cameracontrol system is described, for example, in Japanese Patent ApplicationPublication No. JP-A-2005-354748. Further, a camera control system isknown in which sections corresponding to preset sections are displayedas thumbnails, and when a desired one of the thumbnails is selected, theimaging direction of the camera is controlled to the correspondingpreset section (refer to Japanese Patent Application Publication No.JP-A-2008-22260, for example).

SUMMARY OF THE INVENTION

However, in either of the above-described camera control systemsdescribed in Japanese Patent Application Publication No.JP-A-2005-354748 and Japanese Patent Application Publication No.JP-A-2008-22260, it is not possible to easily ascertain, in the range(the whole image) that can be captured by the camera, which positioncorresponds to the preset section. In addition, in the technologiesdescribed in JP-A-2005-354748 and JP-A-2008-22260, a preset positioncannot be determined on the whole image. Therefore, the imagingdirection cannot be controlled toward the preset section by an operationon the whole image. As a result, it is not possible to intuitivelycontrol the imaging direction to a desired preset direction, and a lotof trial and error is required.

In light of the foregoing, it is desirable to provide a control device,a camera system and a program that are novel and improved and that allowintuitive control of an imaging direction to a preset region that is setin a whole image.

According to an embodiment of the present invention, there is provided acontrol device includes a display control portion that performs controlsuch that a whole image display region, in which a whole image capturedby a camera is displayed, and an enlarged image display region, in whicha region corresponding to a position specified on the whole image isenlarged and an enlarged image is displayed, are displayed, and acontrol portion that performs control such that the region correspondingto the position specified on the whole image is changed. The regionincludes at least one preset region, and the display control portiondisplays, on the whole image, a section that corresponds to the presetregion.

In this configuration, the control portion performs control such thatthe region corresponding to the position specified on the whole image iscaptured by the camera.

In this configuration, when the section that corresponds to the presetregion is selected on the whole image, the control portion controls animaging direction of the camera such that the selected preset region iscaptured.

In this configuration, the display control portion displays, on thewhole image, the section that corresponds to the preset region, based ona size of an angle of view that is set based on the preset region.

In this configuration, when the preset region is not positioned in acenter of the whole image, the display control portion corrects a sizeof a frame at an imaging magnification that is set based on the presetregion, using as a reference a size of the frame when the preset regionis positioned in the center of the whole image, and performs display.

In this configuration, the display control portion changes a position ofthe preset region in accordance with a drag operation performed by auser, and performs display.

In this configuration, when a plurality of the preset regions arespecified on the whole image, the control portion controls an imagingdirection of the camera such that the plurality of the preset regionsare sequentially captured, and the display control portion controlsimage display such that the plurality of the preset regions aredisplayed as a list.

In this configuration, when the plurality of the preset regions arespecified on the whole image, the control portion controls the imagingdirection of the camera such that the plurality of the preset regionsare sequentially captured, and the display control portion alternatelydisplays the plurality of the preset regions.

According to another embodiment of the present invention, there isprovided a camera system includes a camera that captures an image and acontrol device that controls the camera. The control device includes adisplay control portion that performs control such that a whole imagedisplay region, in which a whole image captured by the camera isdisplayed, and an enlarged image display region, in which a regioncorresponding to a position specified on the whole image is enlarged andan enlarged image is displayed, are displayed, and a control portionthat performs control such that the region corresponding to the positionspecified on the whole image is changed. The region includes at leastone preset region, and the display control portion displays, on thewhole image, a section that corresponds to the preset region.

In this configuration, the control portion performs control such thatthe region corresponding to the position specified on the whole image iscaptured by the camera.

In this configuration, when the section that corresponds to the presetregion is selected on the whole image, the control portion controls animaging direction of the camera such that the selected preset region iscaptured.

In this configuration, the display control portion displays, on thewhole image, the section that corresponds to the preset region, based ona size of an angle of view that is set based on the preset region.

In this configuration, when the preset region is not positioned in acenter of the whole image, the display control portion corrects a sizeof a frame at an imaging magnification that is set based on the presetregion, using as a reference a size of the frame when the preset regionis positioned in the center of the whole image, and performs display.

In this configuration, the display control portion changes a position ofthe preset region in accordance with a drag operation performed by auser, and performs display.

In this configuration, when a plurality of the preset regions arespecified on the whole image, the control portion controls an imagingdirection of the camera such that the plurality of the preset regionsare sequentially captured, and the display control portion controlsimage display such that the plurality of the preset regions aredisplayed as a list.

In this configuration, when the plurality of the preset regions arespecified on the whole image, the control portion controls the imagingdirection of the camera such that the plurality of the preset regionsare sequentially captured, and the display control portion alternatelydisplays the plurality of the preset regions.

According to another embodiment of the present invention, there isprovided a program that comprises instructions that command a computerto function as a display control unit which performs control such that awhole image display region, in which a whole image captured by a camerais displayed, and an enlarged image display region, in which at leastone preset region specified on the whole image is enlarged and anenlarged image is displayed, are displayed, and which displays, on thewhole image, a section that corresponds to the preset region, and acontrol unit that performs control such that a region corresponding to aposition specified on the whole image is changed.

According to the present invention described above, it is possible toprovide the control device, the camera system and the program that allowintuitive control of the imaging direction to the preset region that isset in the whole image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing an overall configuration of animaging system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram showing, in detail, configurations of acamera terminal device and a center server in the imaging system shownin FIG. 1;

FIGS. 3A and 3B are schematic diagrams illustrating a pan operation anda tilt operation of the camera terminal device 200;

FIG. 4 is a schematic diagram showing display states of a panorama imageand a bird's eye view image;

FIG. 5 is a schematic diagram illustrating processing to generate abird's eye view image signal DF;

FIG. 6 is a schematic diagram illustrating the processing to generatethe bird's eye view image signal DF;

FIG. 7 is a schematic diagram illustrating the processing to generatethe bird's eye view image signal DF;

FIG. 8 is a schematic diagram illustrating the processing to generatethe bird's eye view image signal DF;

FIG. 9 is a schematic diagram showing display screens of a monitor;

FIGS. 10A, 10B and 10C are schematic diagrams illustrating operations onan operation panel;

FIG. 11 is a schematic diagram showing a state in which menus such as“Screen Mode”, “View Size”, “Image Codec” and “Frame Rate” are displayedwhen a panel “View” is clicked;

FIG. 12 is a schematic diagram showing a menu that is displayed when apanel “Camera” is clicked;

FIG. 13 is a schematic diagram showing respective menus that aredisplayed when “Preset Position”, “Trigger”, “Other” and “Information”are clicked;

FIG. 14 is a schematic diagram illustrating display of the bird's eyeview image;

FIG. 15 is a schematic diagram illustrating display of the bird's eyeview image;

FIG. 16 is a schematic diagram illustrating display of the bird's eyeview image;

FIG. 17 is a schematic diagram illustrating display of the bird's eyeview image;

FIG. 18 is a schematic diagram illustrating display of the bird's eyeview image;

FIG. 19 is a schematic diagram showing a case in which the cameraterminal device according to the present embodiment is provided in aconference room;

FIG. 20 is a schematic diagram showing a state in which a live image isspecified from among four preset positions by selecting one of frames Ato D in the whole image shown in FIG. 19;

FIG. 21 is a schematic diagram showing a state in which a live image isspecified from among the four preset positions by selecting one of theframes A to D in the whole image shown in FIG. 19;

FIG. 22 is a schematic diagram showing a state in which a live image isspecified from among the four preset positions by selecting one of theframes A to D in the whole image shown in FIG. 19;

FIG. 23 is a schematic diagram showing a state in which a live image isspecified from among the four preset positions by selecting one of theframes A to D in the whole image shown in FIG. 19;

FIG. 24 is a schematic diagram showing a method for changing the presetposition;

FIG. 25 is a schematic diagram showing the method for changing thepreset position;

FIG. 26 is a schematic diagram showing a state in which two frames,namely, the frame B and the frame D are selected by a drag operationperformed by a user;

FIG. 27 is a schematic diagram showing an example in which an imagecorresponding to the frame B and an image corresponding to the frame Dare both displayed when the two frames (the frame B and the frame D) areselected by the drag operation performed by the user; and

FIG. 28 is a schematic diagram showing, in detail, an example of asetting screen to set the preset position.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Note that the description will be made in the following order.

1. First embodiment (example of imaging system including camera terminaldevices and servers)

-   -   (1) Configuration example of imaging system    -   (2) Configuration example of camera terminal device and center        server    -   (3) Display operations of first panorama image and second        panorama image    -   (4) Processing to generate first panorama image    -   (5) Graphical user interface (GUI)    -   (6) Specifying of imaging direction on first panorama image    -   (7) Control using preset position

1. First Embodiment (1) Configuration Example of Imaging System

FIG. 1 is a schematic diagram showing an overall configuration of animaging system 100 according to an embodiment of the present invention.The imaging system 100 includes camera terminal devices (IP cameras)200, a center server 300 and clients 400. A plurality of the cameraterminal devices 200, the center server 300 and the clients 400 areconnected via a network 500. A proxy server 600 is connected to thenetwork 500, and a plurality of the camera terminal devices 200 areconnected to the proxy server 600. Further, a monitor 320 is connectedto the center server 300. In the present embodiment, the camera terminaldevices 200 are, for example, monitoring cameras that are placed indoorsor outdoors. A region in which each of the camera terminal devices 200is provided can be monitored by displaying, on the monitor 320 of thecenter server 300, an image acquired by each of the camera terminaldevices 200. The clients 400 are connected, via the network 500, to thecamera terminal devices 200, the center server 300 or the proxy server600. The clients 400 may have a function of the center server 300 and afunction of the monitor 320, which will be described later. Imagescaptured by the camera terminal devices 200 are transmitted to theclients 400 via the network 500, and a first panorama image and a secondpanorama image are displayed on a display screen of each of the clients400 together with a live image that will be described later. Note that,although the IP cameras are shown as an example of the camera terminaldevices 200, the camera terminal devices 200 are not limited to the IPcameras and they may be analog cameras.

Panorama images (a full view image, a whole image) indicate a range thatcan be captured by each of the camera terminal devices 200. There arevarious shapes of panorama images, examples of which include a circularpanorama image and a rectangular panorama image such as those shown inFIG. 4, and a loop-shaped panorama image as projected onto a cylindricalsurface. Hereinafter, in the present embodiment, the circular panoramaimage and the rectangular panorama image are explained as examples ofthe first panorama image and the second panorama image, respectively.

(2) Configuration Example of Camera Terminal Device and Center Server

FIG. 2 is a schematic diagram showing, in detail, configurations of thecamera terminal device 200 and the center server 300 in the imagingsystem shown in FIG. 1. Respective structural elements of the cameraterminal device 200 or the center server 300 shown in FIG. 2 can beformed by hardware (circuits or the like), or by an arithmeticprocessing unit, such as a CPU, and software (a program) that causes thearithmetic processing unit to function. A camera portion 201 of thecamera terminal device 200 includes an imaging optical system. Thecamera portion 201 performs an imaging operation based on an imagingcontrol signal CTa that is supplied from a terminal control portion 210to be described later, and generates a video signal Dcam. Further, thecamera portion 201 supplies the generated video signal Dcam to a signalprocessing portion 202 via a bus 220. Note that a memory portion 203, animaging direction control portion 204, a network interface portion 206,a panorama image storage portion 207 and the terminal control portion210 are also connected to the bus 220.

The signal processing portion 202 performs compression processing of thevideo signal Dcam, and stores an obtained video coding signal DV in thememory portion 203. Further, the signal processing portion 202 uses thevideo signal Dcam, which is obtained by sequentially moving the imagingdirection of the camera portion 201, to generate a video signal DF ofthe first panorama image (this signal is hereinafter also referred to asa first panorama image signal DF) and a video signal DP of the secondpanorama image (this signal is hereinafter also referred to as a secondpanorama image signal DP). Then, the signal processing portion 202stores the generated first panorama image signal DF and second panoramaimage signal DP in the panorama image storage portion 207. Note that thecompression processing of the video signal Dcam and the generation ofthe first panorama image signal DF and the second panorama image signalDP are performed based on a signal processing control signal CTbsupplied from the terminal control portion 210 that will be describedlater.

In order to move the imaging direction of the camera portion 201 to adirection instructed by a direction control signal CTc supplied from theterminal control portion 210 that will be described later, the imagingdirection control portion 204 calculates speed and acceleration whenmoving the imaging direction. Further, the imaging direction controlportion 204 generates, based on a calculation result, a driving signalMDp to perform a pan operation and a driving signal MDt to perform atilt operation, and supplies the generated driving signals MDp and MDtto an imaging direction adjustment portion 205. Further, the imagingdirection control portion 204 generates a camera position informationsignal PM that indicates which direction is the imaging direction of thecamera portion 201, and supplies the generated camera positioninformation signal PM to the terminal control portion 210.

The imaging direction adjustment portion 205 includes a pan operationmotor that causes the camera portion 201 to move in the left-rightdirection, and a tilt operation motor that causes the camera portion 201to move in the up-down direction. The imaging direction adjustmentportion 205 drives the pan operation motor using the driving signal MDp,and drives the tilt operation motor using the driving signal MDt. Thus,the imaging direction adjustment portion 205 adjusts the imagingdirection of the camera portion 201 to the direction instructed by thedirection control signal CTc. Note that it is assumed that the imagingdirection adjustment portion 205 performs a continuous pan operationwithout limitation on its movement range.

FIG. 3 is a schematic diagram illustrating a pan operation and a tiltoperation of the camera terminal device 200, and shows a case in whichthe camera terminal device 200 is attached to a ceiling or the like.FIG. 3B shows a pan/tilt operation of the camera terminal device 200according to the present embodiment. On the other hand, FIG. 3A shows,for comparison, a pan/tilt operation of a general camera terminaldevice. As shown in FIG. 3A, in the general camera terminal device, acontinuous 360-degree pan operation is performed, and the tilt operationis performed over a range of degrees from a vertical direction to ahorizontal direction. In contrast, in the camera terminal device 200according to the present embodiment, a continuous 360-degree panoperation is performed, and the tilt operation is performed over a rangeof 220 degrees centered on the vertical direction. Accordingly, thevideo signal DF of the first panorama image can be generated using atechnique that will be described later and can be stored in the panoramaimage storage portion 207. The first panorama image that will bedescribed later is a circular panorama image. When the imaging directionis operated across the center in the vicinity of the center of thecircle, the camera terminal device 200 according to the presentembodiment can also mechanically move to a target imaging direction inthe shortest distance. In contrast, in the general camera terminaldevice, it is not possible to mechanically move the camera imagingdirection across the center of the circle.

The network interface portion 206 is an interface to performcommunication between the camera terminal device 200 and the centerserver 300 via the network 500.

The terminal control portion 210 controls the camera portion 201 usingthe imaging control signal CTa, controls the signal processing portion202 using the signal processing control signal CTb, and controls theimaging direction control portion 204 using the direction control signalCTc. Thus, the terminal control portion 210 performs image capture whilesequentially moving the imaging direction. Then, based on the obtainedvideo signal Dcam, the terminal control portion 210 generates the firstpanorama image signal DF and the second panorama image signal DP andstores them in the panorama image storage portion 207.

The imaging system 100 according to the present embodiment can transmit,from the camera terminal device 200 to the center server 300, a liveimage generated based on the video signal Dcam, together with the firstpanorama image generated based on the first panorama image signal DF orthe second panorama image generated based on the second panorama imagesignal DP, and can display them on the monitor 320. Note that, in thisspecification, there are cases in which the first panorama image or thesecond panorama image is referred to as the whole image. FIG. 4 is aschematic diagram showing display states of the second panorama imageand the first panorama image. The center server 300 can display on themonitor 320 one of the first panorama image and the second panoramaimage in accordance with a user selection, together with the live image.Further, the center server 300 may be structured such that it displaysboth the first panorama image and the second panorama image togetherwith the live image. Note that, the circular panorama image on the rightside of FIG. 4, in which an image in the vicinity of the center ismissing, is shown as the first panorama image. However, the firstpanorama image is not limited to this example, and a circular panoramaimage in which the image in the vicinity of the center is not missingmay be used as the first panorama image.

The camera terminal device 200 is placed on a ceiling or the like, anddisplays an image in a downward direction from the camera terminaldevice 200. Therefore, as shown in FIG. 4, the first panorama image isacquired over a range of pan 360 degrees and tilt 220 degrees, centeredon a vertical line extending downward from the camera terminal device200, and the contour of the first panorama image is circular.

Further, the imaging direction adjustment portion 205 is structured suchthat it can perform the pan operation continuously as described above.Therefore, it is assumed that the second panorama image that is storedin the panorama image storage portion 207 is formed such that the imagedoes not overlap as shown in FIG. 4. For example, it is assumed that oneend of the second panorama image has an angle difference of “+180degrees” from a reference direction, and the other end of the secondpanorama image has an angle difference of “−180 degrees” from thereference direction. Further, since an operation range of the panoperation is not limited, a center position of the operation rangecannot be taken as the reference direction, in contrast to a case inwhich the operation range is limited, for example. Accordingly, thereference direction is set in advance and the second panorama imagesignal DP is generated such that this reference direction corresponds tothe center position of the second panorama image. The imaging directioncontrol portion 204 generates, as the camera position information signalPM, a signal indicating, for example, an angle difference of the imagingdirection with respect to the reference direction set in advance.

If the reference direction is set in advance in this manner, there is noneed to determine which direction is taken as the reference in animaging operation to generate the second panorama image. Further, if theangle difference of the imaging direction with respect to the referencedirection is indicated by the camera position information signal PM, theposition of the imaging direction based on the camera positioninformation signal PM can be easily determined on the second panoramaimage.

Also in the tilt operation, it is assumed that the reference directionis set in advance and the angle difference of the imaging direction withrespect to the reference direction is indicated by the camera positioninformation signal PM. Thus, both on the first panorama image and thesecond panorama image, the position of the imaging direction based onthe camera position information signal PM can be easily determined. Notethat, in the tilt operation, when the camera terminal device 200 isplaced on the ceiling, the vertical direction can be taken as thereference direction.

The terminal control portion 210 analyses a command signal CM that issupplied from the center server 300 via the network interface portion206, and generates the imaging control signal CTa, the signal processingcontrol signal CTb and the direction control signal CTc. Further, theterminal control portion 210 performs processing to transmit, to thecenter server 300, the video coding signal DV stored in the memoryportion 203, the first panorama image signal DF and the second panoramaimage signal DP that are stored in the panorama image storage portion207, and the camera position information signal PM supplied from theimaging direction control portion 204.

A network interface portion 301 of the center server 300 is an interfaceto perform communication between the camera terminal device 200 and thecenter server 300 via the network 500. The network interface portion 301supplies the video coding signal DV supplied from the camera terminaldevice 200 to an extension processing portion 302. Further, the networkinterface portion 301 supplies the camera position information signal PMto a position determination processing portion 303, and supplies thefirst panorama image signal DF and the second panorama image signal DPto an image processing portion 304.

The extension processing portion 302 performs extension processing ofthe video coding signal DV and supplies the obtained video signal Dcamto a display processing portion 305.

The position determination processing portion 303 determines whichposition in the first panorama image or the second panorama imagecorresponds to the imaging direction indicated by the camera positioninformation signal PM, and supplies a position determination result KPto the image processing portion 304. Here, if it is assumed that thecamera position information signal PM generates the first panorama imagesignal DF or the second panorama image signal DP such that the referencedirection set in advance as described above is the center position ofthe pan operation and the tilt operation and such that the cameraposition information signal PM indicates an angle difference between thereference direction and a current imaging direction, it is easilydetermined, based on the camera position information signal PM, whichposition in a bird's eye view image (i.e., the first panorama image) orin the second panorama image corresponds to the current imagingdirection.

The image processing portion 304 performs image processing based on theposition determination result KP, and generates a first panorama imagesignal DFC of the first panorama image and a second panorama imagesignal DPC of the second panorama image. Regarding the second panoramaimage signal DPC, it is possible to generate the second panorama imagesignal DPC in which the position determined by the positiondetermination result KP is set as the center position. In this case, theimage processing portion 304 determines a difference amount between thedetermined position and the center position of the second panoramaimage. Then, based on the determined difference amount, the secondpanorama image is processed such that the determined position becomesthe center position of the image. More specifically, a display imageframe of the second panorama image of “±180 degrees” is set with thedetermined position being the center position, and the second panoramaimage of a region corresponding to the difference amount that deviatesfrom the display image frame of the second panorama image is pasted to aregion with no image. Thus, the second panorama image signal DPC of thesecond panorama image in which the determined position is set as thecenter position is generated. Also in the case of the first panoramaimage signal DFC, when in a “mode for rotating the first panorama image”that will be described later, in a similar manner to the second panoramaimage signal DPC, a difference amount between a determined position anda reference position in the pan direction of the first panorama image isdetermined, and based on the determined difference amount, the firstpanorama image is processed such that the determined position is placedon the image. Note that, in a case where the position determined by theposition determination result KP is not set as the center position, theimage processing portion 304 generates the first panorama image signalDFC of the first panorama image and the second panorama image signal DPCof the second panorama image, based on the first panorama image signalDF and the second panorama image signal DP, respectively, withoutchanging the center position. Further, the image processing portion 304displays a position based on the position determination result KP (i.e.,a position toward which an optical axis of the imaging optical system ofthe camera portion 201 is directed) in each of the first panorama imagesignal DFC and the second panorama image signal DPC. Further, the imageprocessing portion 304 supplies, to the display processing portion 305,the first panorama image signal DFC and the second panorama image signalDPC that have been generated by performing the image processing.

The display processing portion 305 generates a display driving signal HDusing the video signal Dcam supplied from the extension processingportion 302 and the first panorama image signal DFC and the secondpanorama image signal DPC that are supplied from the image processingportion 304, and supplies the generated display driving signal HD to themonitor 320. Further, the display processing portion 305 generates thedisplay driving signal HD using a GUI video signal that is generatedbased on a GUI display control signal CTg supplied from the controlportion 310 that will be described later, and supplies the generateddisplay driving signal HD to the monitor 320. The display processingportion 305 can perform control such that live images and panoramaimages transmitted from the plurality of camera terminal devices 200 aredisplayed as a list on the monitor 320.

The monitor 320 drives a display element (for example, a liquid crystaldisplay element or a plasma display element), a cathode-ray tube or thelike based on the display driving signal HD, and thereby displays on thescreen the first panorama image, the second panorama image in which theimaging direction is set to the center position, an image (a live image)captured by the camera portion 201, a GUI image and the like.

A GUI is used in a user interface portion 315, and the user interfaceportion 315 presents information to the user using the monitor 320. Whena user operation is performed using an operation input device, such as apointing device, a keyboard or the like, based on the presentedinformation, the user interface portion 315 supplies an operation signalUS in accordance with the user operation to the control portion 310, andcauses a desired operation to be performed.

Based on the operation signal US and a display state of the monitor 320,the control portion 310 determines which processing the user hasselected or which processing the user has requested. Then, the controlportion 310 generates a control signal CTm based on a determinationresult, and supplies the generated control signal CTm to each portion,thereby controlling an operation of the center server 300. Further, thecontrol portion 310 generates the command signal CM based on thedetermination result, and supplies the generated command signal CM tothe camera terminal device 200 via the network interface portion 301,thereby controlling an operation of the camera terminal device 200.Further, the control portion 310 generates the GUI display controlsignal CTg and supplies it to the display processing portion 305.

In the above-described examples, the camera terminal device 200generates the video signal DF of the first panorama image and the videosignal DP of the second panorama image from the video signal Dcam.However, the center server 300 or the client 400 may generate the videosignal DF of the first panorama image and the video signal DP of thesecond panorama image. In this case, the network interface portion 206of the camera terminal device 200 transmits the video signal Dcam outputfrom the camera portion 201 to the center server 300 or to the client400. If the center server 300 or the client 400 is provided with asignal processing portion similar to that of the camera terminal device200, the center server 300 or the client 400 can generate the videosignal DF of the first panorama image and the video signal DP of thesecond panorama image. The video signal DF of the first panorama imageand the video signal DP of the second panorama image are stored in amemory, such as a hard disk, provided in the center server 300 or theclient 400.

(3) Display Operations of First Panorama Image and Second Panorama Image

Next, display operations of the first panorama image and the secondpanorama image will be explained. When an operation is started, thecontrol portion 310 transmits to each of the camera terminal devices 200the command signal CM that requests transmission of the first panoramaimage signal DF, the second panorama image signal DP, the cameraposition information signal PM and the video coding signal DV. When thefirst panorama image signal DF and the second panorama image signal DPare stored in the panorama image storage portion 207, the cameraterminal device 200 reads out the first panorama image signal DF and thesecond panorama image signal DP and transmits them to the center server300. On the other hand, when the first panorama image signal DF or thesecond panorama image signal DP is not stored in the panorama imagestorage portion 207, the camera terminal device 200 controls the cameraportion 201, the signal processing portion 202 and the imaging directioncontrol portion 204, and performs an imaging operation while moving theimaging direction. Thus, the camera terminal device 200 generates thefirst panorama image signal DF and the second panorama image signal DP,and transmits them to the center server 300. Note that the cameraterminal device 200 stores the generated first panorama image signal DFand the second panorama image signal DP in the panorama image storageportion 207.

The camera terminal device 200 supplies the camera position informationsignal PM generated in the imaging direction control portion 204 to thecenter server 300. Further, the camera terminal device 200 starts theimaging operation of the camera portion 201, compresses the obtainedvideo signal Dcam in the signal processing portion 202, and stores thevideo coding signal DV in the memory portion 203. Further, the cameraterminal device 200 supplies the video coding signal DV stored in thememory portion 203 to the center server 300. Note that, when the cameraterminal devices 200 and the network 500 are connected via the proxyserver 600 as shown in FIG. 1, information is transmitted and receivedbetween each of the camera terminal devices 200 and the center server300 via the proxy server 600.

The position determination processing portion 303 of the center server300 determines which position in the first panorama image or the secondpanorama image corresponds to the imaging direction indicated by thecamera position information signal PM, and supplies the positiondetermination result KP to the image processing portion 304. If thesecond panorama image is taken as an example, when, for example, theangle difference indicated by the camera position information signal PMis “0 degrees”, because the imaging direction is set to the referencedirection, the position of the imaging direction is the center positionof the second panorama image.

The image processing portion 304 performs the image processing based onthe position determination result KP. When the position determinationresult KP indicates that the position of the imaging direction is thecenter position of the second panorama image, an image in the imagingdirection corresponds to the center of the second panorama image.Therefore, the image processing of the second panorama image is notperformed, and the second panorama image signal DP is supplied to thedisplay processing portion 305 as the second panorama image signal DPC.On the other hand, when the position of the imaging direction is not thecenter position of the second panorama image, a difference amountbetween the determined position of the imaging direction and the centerposition of the second panorama image is determined. Then, based on thedetermined difference amount, the second panorama image is processedsuch that the determined position corresponds to the center position ofthe image, and the second panorama image signal DPC is generated.

Therefore, in the second panorama image, when, for example, the angledifference indicated by the camera position information signal PM is “0degrees”, an image captured by the camera portion 201, the secondpanorama image in which the reference direction corresponds to thecenter position, and an image relating to the GUI are displayed on thescreen of the monitor 302 that is driven by the display driving signalHD supplied from the display processing portion 305.

In the case of the first panorama image, when in the “mode for rotatingthe first panorama image” that will be described later, in a similarmanner to the second panorama image signal DPC, a difference amountbetween the determined position and the reference position in the pandirection of the first panorama image is determined. Then, based on thedetermined difference amount, the first panorama image is rotated suchthat the determined position is placed below the image (on a center lineL to be described later), and the first panorama image signal DFC isgenerated. The image processing portion 304 displays the position (i.e.,the position toward which the optical axis of the imaging optical systemof the camera portion 201 is directed) of the imaging direction based onthe position determination result KP, in each of the first panoramaimage signal DFC and the second panorama image signal DPC.

In either of the first panorama image and the second panorama image, thereference position set in advance can be fixed and displayed as thecenter position. In this case, a difference amount between the positiondetermined by the position determination result KP and the referenceposition set in advance is determined, and based on the determineddifference amount, a cursor, a frame or the like, which indicates thatthe imaging direction (the direction of the optical axis of the opticalsystem included in the camera portion 201) of the camera is directed toa position that is removed from the center position by the differenceamount, is displayed in the first or second panorama image.

The extension processing portion 302 extends the video coding signal DV,and supplies the obtained video signal Dcam to the display processingportion 305. The control portion 310 supplies the GUI display controlsignal CTg for performing GUI display to the display processing portion305.

(4) Processing to Generate First Panorama Image

Next, processing to generate the first panorama image signal DF will beexplained with reference to FIG. 5 to FIG. 8. When the first panoramaimage signal DF is generated, an image is acquired for each region ofthe first panorama image. First, as shown in FIG. 5, the zoom of thecamera terminal device 200 is fixed, and a pan/tilt operation isperformed at given intervals and still images are acquired. Numbersshown in FIG. 5 indicate the order of image acquisition.

Next, coordinate conversion of a still image coordinate system isperformed by the following procedures [1] to [3]. Here, as shown in FIG.6, each point of the still image is converted to a relative position (x,y, z) as viewed from the center of rotation of the camera terminaldevice 200.

[1] The point of view of the camera with a horizontal angle of view ofthh [degrees] and an aspect ratio of “a” is set at an origin and isdirected to a -z direction. If a subject plane (x, y, -L) is consideredat the position of a distance L, a visible range is a rectangle of2Lx×2Ly. Here, the following relationship is established: Lx=L·tan(thh/2), Ly=Lx/a.

[2] The posture of the camera is originally (x, y, z, p, q, r). However,considering that the position of the point of view is fixed at theorigin, the equation x=y=z=0 is obtained. Therefore, here, only (p, q,r) are considered (p: pan (y axis rotation), q: tilt (x axis rotation),r: roll (z axis rotation)).

[3] When a point (j, i) in the rectangle of (2Lx)×(2Ly) is used, thefollowing equations are established.

x=Lx×(j−Lx)/Lx

y−Ly×(i−Ly)/Ly

z=−L

[4] The following equations of roll conversion, tilt conversion and panconversion are applied using (x, y, z) .

$\begin{matrix}{{{{Roll}\mspace{14mu} {{conversion}\begin{bmatrix}{x\; 1} \\{y\; 1} \\{z\; 1}\end{bmatrix}}} = {\begin{bmatrix}{\cos \; r} & {{- \sin}\; r} & 0 \\{\sin \; r} & {\cos \; r} & 0 \\0 & 0 & 1\end{bmatrix} \cdot \begin{bmatrix}x \\y \\z\end{bmatrix}}}{{{Tilt}\mspace{14mu} {{conversion}\begin{bmatrix}{x\; 2} \\{y\; 2} \\{z\; 2}\end{bmatrix}}} = {\begin{bmatrix}1 & 0 & 0 \\0 & {\cos \; q} & {{- \sin}\; q} \\0 & {\sin \; q} & {\cos \; q}\end{bmatrix} \cdot \begin{bmatrix}{x\; 1} \\{y\; 1} \\{z\; 1}\end{bmatrix}}}{{{Pan}\mspace{14mu} {{conversion}\begin{bmatrix}{x\; 3} \\{y\; 3} \\{z\; 3}\end{bmatrix}}} = {\begin{bmatrix}{\cos \; p} & 0 & {{- \sin}\; p} \\0 & 1 & 0 \\{\sin \; p} & 0 & {\cos \; p}\end{bmatrix} \cdot \begin{bmatrix}{x\; 2} \\{y\; 2} \\{z\; 2}\end{bmatrix}}}} & {{Expression}\mspace{14mu} 1}\end{matrix}$

Values of the point (x3, y3, z3) obtained here correspond to valuesexpressed in a coordinate system when a point (x, y, z) in the stillimage is viewed from the center of rotation of the camera.

Next, a relative position is converted to a latitude and longitude.Here, as shown in FIG. 7, the point (x3, y3, z3) obtained as describedabove is converted to a latitude and longitude. An angle between adirection of the point (x3, y3, z3) and a horizontal direction withrespect to the −z direction is the longitude, and an angle between thedirection of the point (x3, y3, z3) and an x-z plane is the latitude,and the latitude and longitude can be obtained by the followingequations.

-   -   Latitude (La)

La=tan−1(y3/::sqrt(x3*x3+z3*z3))

-   -   Longitude (Lo)    -   when 0.0<z3,

if 0.0<x3, Lo=90.0+La

if 0.0≧x3, Lo−−90.0−La

-   -   when 0.0≧z3,

Lo=tan-1(x 3/z3)/RAD

where RAD=n/180(n:circle ratio)

Next, conversion to the first panorama image is performed. Here, asshown in FIG. 8, the obtained latitude and longitude (La, Lo) areexpressed such that the latitude is indicated by a radial axis and thelongitude is indicated by a circumferential coordinate system.

In the manner described above, the first panorama image signal DF can begenerated.

(5) Graphical User Interface (GUI)

Next, the GUI will be explained. FIG. 9 is a schematic diagram showingdisplay screens of the monitor 320. FIG. 9 shows both the normal screendisplay and the full screen display. In either case, an operation panel(a control panel) is displayed on the left side of the screen and a liveimage is displayed on the right side of the screen. When a “Screen Mode”of the operation panel is set to “Full”, the full screen display isshown.

FIG. 10 is a schematic diagram illustrating operations on the operationpanel. The operation panel includes panels for each function, and eachof the panels can be folded and stored. The panels can be classified inthe following manner.

-   -   View Settings relating to display    -   Camera control        -   Settings relating to camera operation    -   Panorama        -   Panorama display    -   Preset position        -   Settings relating to preset    -   Trigger        -   Settings relating to trigger    -   Other        -   Other settings

When each of the panels is clicked, the clicked panel is opened. FIG.10A shows a state in which the panel “View” is clicked, and a menu isdisplayed below a “View” column. On the other hand, FIG. 10B indicates astate in which the panel “Camera control” is clicked, and a menu isdisplayed below a “Camera control” column. FIG. 10C indicates a state inwhich the panel “Camera control” is dragged, and the panel “Cameracontrol” can be changed to a floating window by dragging it.

FIG. 11 shows a state in which menus such as “Screen Mode”, “View Size”,“Image Codec” and “Frame Rate” are displayed when the panel “View” isclicked. With the “Screen Mode” menu, the display mode can be changed toa normal mode and a full screen mode. With the “View Size” menu, ¼,½,the same size as an image, and a fit mode can be specified. In the fitmode, an image is displayed in accordance with the size of a currentdisplay area. With the “Image Codec” menu, a codec change can beperformed. With the “Frame Rate” menu, a JPEG frame rate can be changed.A moving image recording button is used to record moving images, and astill image recording button is used to record still images. In additionto these buttons, a microphone volume slider, a microphone mute on/offbutton, an audio volume slider, an audio mute on/off button, and thelike are provided.

FIG. 12 shows a menu that is displayed when a panel “Camera” is clicked.With an “Operation Mode” menu, operation mode switching can be performedon an image. Pan/tilt directions can be specified by operating pan/tiltcontrol buttons with an arrow on them. A button located in the center ofthe pan/tilt control buttons is a home position button. A zoom button isa button to specify the zoom of the camera terminal device 200, and itis formed such that a boundary between an optical zoom and a digitalzoom can be seen. If “W” or “T” of the zoom button is pressedcontinuously, a zoom operation is performed continuously. A focus buttonis used to set the focus to a “long distance” or a “short distance”. Inaddition to these buttons, various kinds of buttons are provided, suchas a one push auto focus button and a control right acquisition buttonfor exclusive control.

FIG. 13 shows respective menus that are displayed when “PresetPosition”, “Trigger”, “Other” and “Information” are clicked. The panel“Preset Position” displays thumbnail images in a plurality of imagingdirections determined in advance. By clicking and selecting one of thethumbnail images, it is possible to direct the camera terminal device200 in a direction of the selected thumbnail image.

(6) Specifying of Imaging Direction on First Panorama Image

Next, display of the first panorama image will be explained withreference to FIG. 14 to FIG. 18. FIG. 14 is a schematic diagram showinga state in which the first panorama image is displayed by clicking“Preset position” of the operation panel. Note that a live image isdisplayed on the right side of the operation panel. As described above,by dragging and moving the “Preset position” column, a sub-panel thatdisplays the first panorama image can be arranged outside of theoperation panel as shown in FIG. 15. Further, the sub-panel thatdisplays the first panorama image can be moved within the displayscreen.

FIG. 16 is a schematic diagram showing, in detail, the sub-panel thatdisplays the first panorama image. In the first panorama image, thedirection (the direction of the optical axis of the imaging opticalsystem of the camera terminal device 200) of the image displayed in thelive image on the right side is shown by a cross (+). Further, as shownin FIG. 16, the first panorama image can be switched to the secondpanorama image by clicking a panorama switching button arranged on theupper right side of the first panorama image.

FIG. 17 shows an operation to change the direction of the live imageusing the first panorama image. When the pan/tilt direction of thecamera terminal device 200 is changed, the above-described pan/tiltcontrol button can be used for the change. However, the pan/tiltdirection of the camera terminal device 200 can also be changed byspecifying a given point in the first panorama image. In the state shownin FIG. 17, the optical axis of the imaging optical system of the cameraterminal device 200 is directed toward the direction of the cross (+)and the live image in that direction is displayed. In this state, if ablack circle () shown in the first panorama image is clicked, theoperation signal US in accordance with a user operation is transmittedto the control portion 310. Based on the operation signal US, theimaging direction control portion 204 of the camera terminal device 200is controlled, and the optical axis of the imaging optical system of thecamera terminal device 200 is directed toward the direction of the blackcircle (). Accordingly, the live image displayed on the right side ofthe operation panel is switched to an image in the direction of theblack circle ().

Thus, the user can specify the pan/tilt direction of the camera terminaldevice 200 by specifying a given point in the first panorama image. Thefirst panorama image includes all the images in the pan/tilt directionsthat can be captured by the camera terminal device 200. Therefore, theuser can display the live image in a desired direction by specifying thedirection of the camera terminal device 200 on the first panorama image.Note that the direction of the optical axis may be controlled byspecifying a region using a frame that surrounds a predetermined range,instead of using the cross (+) that indicates a click position.

When the direction of the camera terminal device 200 is specified on thefirst panorama image, there are two modes, namely, a mode in which thedisplay state of the first panorama image is not changed and only thelive image is switched, and a mode in which the first panorama imagerotates. In the mode in which the first panorama image rotates, as shownin FIG. 18, an angle position of the first panorama image is set suchthat the position (indicated by the cross (+)) at which the live imageis currently being displayed is positioned on a straight line Lextending upward from the center of the first panorama image. Next, whenthe black circle () is clicked on the first panorama image, the liveimage displayed on the right side of the operation panel is switched toan image in the direction of the black circle (). At the same time, thefirst panorama image rotates such that the position of the black circle() is located on the straight line L in the first panorama image.Accordingly, when the user looks at the first panorama image, the usercan immediately recognize that the direction of the live image is on thestraight line L of the first panorama image.

In a similar manner, in the second panorama image also, the user canspecify the pan direction of the camera terminal device 200 byspecifying a given point in the second panorama image. Based on this,the user can switch the direction of the live image.

(7) Control Using Preset Position

Next, control using a preset position in the system of the presentembodiment will be explained. As described above, in the system of thepresent embodiment, by specifying a desired region in the first panoramaimage or the second panorama image, the imaging direction of the cameraterminal device 200 can be directed toward the specified direction. Onthe other hand, when one or a plurality of preset positions are set inadvance, by selecting the preset position (one of the preset positions),the imaging direction of the camera terminal device 200 can be directedtoward the selected position.

FIG. 19 shows, as an example, a case in which the camera terminal device200 according to the present embodiment is provided in a conferenceroom. In the example shown in FIG. 19, four preset positions are set forimage capture by the camera terminal device 200. At each of the fourpreset positions, a “whiteboard”, a “projection screen”, a “relativelywide range of the conference room” and a “specific of the conferenceroom” are captured. Further, in a lower part of the screen, a wholeimage is displayed as the second panorama image.

Ranges corresponding to the respective preset positions are displayed inthe second panorama image using frames A to D. Here, the frame Acorresponds to the “whiteboard”, the frame B corresponds to the“relatively wide range of the conference room”, the frame C correspondsto the “projection screen”, and the frame D corresponds to the“relatively narrow range of the conference room”.

Since the frames of the imaging ranges corresponding to the respectivepreset positions are displayed in this manner, the user can visually anddirectly determine which range in the panorama image corresponds to eachof the preset positions.

The preset positions are set in advance by the user using a method thatwill be described later. Data of the preset positions is stored in thepanorama image storage portion 207 of the camera terminal device 200, ora memory provided in the center server 300, a memory provided in theclient 400 or the like.

Next, a method for displaying, from the panorama image, a live image ofeach of the preset positions set as described above will be explained.FIG. 20 to FIG. 23 are schematic diagrams each showing a state in whicha live image is specified from among the four preset positions byselecting one of the frames A to D in the whole image shown in FIG. 19.

FIG. 20 shows a state in which the frame A in the whole image isselected from the state shown in FIG. 19. The selection of the frame Acan be performed by moving a cursor on the display screen to theposition of the frame A in the whole image and clicking the frame A. Asshown in FIG. 20, when the frame A in the whole image is clicked, thelive image “whiteboard” corresponding to the frame A is displayed in alarge size on the whole image.

In a similar manner, FIG. 21 shows a state in which the frame B in thewhole image is clicked. FIG. 22 shows a state in which the frame C inthe whole image is clicked. FIG. 23 shows a state in which the frame Din the whole image is clicked. As shown in FIG. 21 to FIG. 23, when theframe B, the frame C and the frame D are clicked, the live imagesrespectively corresponding to the frame B, the frame C and the frame Dare displayed. Accordingly, the user can switch and display a desiredone of the live images, by selecting one of the frame A, the frame B,the frame C and the frame D while looking at the whole image.

In this case, when the user clicks one of the frames A to D, theoperation signal US in accordance with the user operation is transmittedfrom the user interface portion 315 to the control portion 310. Theoperation signal US includes information of the selected one of theframes A to D (namely, the selected one of the preset positions). Thecontrol portion 310 transmits the command signal CM, which is generatedbased on the operation signal US, to the terminal control portion 210 ofthe camera terminal device 200 via the network interface portion 206.The terminal control portion 210 analyzes the command signal CM, andgenerates the direction control signal CTc using the information of thepreset position stored in the panorama image storage portion 207. Thus,the imaging direction of the camera portion 201 is adjusted, by theimaging direction control portion 204 and the imaging directionadjustment portion 205, to a direction instructed by the directioncontrol signal CTc.

Note that, when the information of the preset position is stored in thememory provided in the center server 300, the information is included inthe command signal CM and is transmitted to the camera terminal device200. Then, the terminal control portion 210 analyzes the command signalCM, and generates the direction control signal CTc using the informationof the preset position transmitted from the center server 300.

FIG. 24 and FIG. 25 are schematic diagrams showing a method for changingthe preset position. Specifically, FIG. 24 and FIG. 25 show a method forchanging the preset position by moving the position of the frame B inthe panorama image. More specifically, if the user drags the frame B inthe state shown in FIG. 24 and ends the drag operation when the frame Bis moved to a position shown in FIG. 25, the user can change the presetposition that corresponds to the frame B to the position shown in FIG.25. In the state shown in FIG. 25, as a result of the drag operationperformed by the user, the position of the frame B is moved to aposition that corresponds to the “whiteboard” and the “projectionscreen”, and a live image of the “whiteboard” and the “projectionscreen” that corresponds to the moved frame B is displayed. Thus, thepreset position corresponding to the frame B is set to the position thatcorresponds to the “whiteboard” and the “projection screen”.

In this case, when the user drags the frame B to move it, the operationsignal US in accordance with the user operation is transmitted from theuser interface portion 315 to the control portion 310. The operationsignal US includes information of the frame B (the preset positioncorresponding to the frame B) moved by the drag operation. The controlportion 310 transmits the command signal CM, which is generated based onthe operation signal US, to the terminal control portion 210 of thecamera terminal device 200 via the network interface portion 206. Theterminal control portion 210 analyzes the command signal CM, and changesthe information of the preset position stored in the panorama imagestorage portion 207. At the same time, the terminal control portion 210generates the direction control signal CTc based on the changed presetposition. Thus, the changed preset position is stored in the panoramaimage storage portion 207, and the imaging direction of the cameraportion 201 is adjusted to a direction instructed by the directioncontrol signal CTc.

Note that, when the information of the preset position is stored in thememory provided in the center server 300, the control portion 310changes the preset position stored in the memory, based on the commandsignal CM generated based on the operation signal US. Then, the controlportion 310 transmits the command signal CM that includes the changedpreset position to the terminal control portion 210 of the cameraterminal device 200 via the network interface portion 206. The terminalcontrol portion 210 analyzes the command signal CM, and generates thedirection control signal CTc based on the changed preset position. Thus,the imaging direction of the camera portion 201 is adjusted to thedirection instructed by the direction control signal CTc.

In this manner, by dragging and moving each of the frames displayed inthe panorama image, it is possible to freely change the preset position.In this case, all the ranges that can be captured by the camera terminaldevice 200 are displayed on the panorama image. Therefore, the user canvisually set a desired one of the preset positions from among all theranges that can be captured by the camera terminal device 200.

Next, an example will be explained in which a plurality of live imagesthat are set in the panorama image are displayed. FIG. 26 shows a statein which two frames, namely, the frame B and the frame D are selected bya drag operation performed by the user. In this case, the image of the“relatively wide range of the conference room” that corresponds to theframe B and the image of the “relatively narrow range of the conferenceroom” that corresponds to the frame D are alternately displayed as liveimages at predetermined time intervals.

FIG. 27 shows a case in which, when the two frames, namely, the frame Band the frame D are selected by a drag operation performed by the user,the image of the “relatively wide range of the conference room” thatcorresponds to the frame B and the image of the “relatively narrow rangeof the conference room” that corresponds to the frame D are bothdisplayed. In this case, since two live images cannot be displayedsimultaneously, the two images are alternately updated to latest liveimages. Alternatively, one of the two images may be displayed as a liveimage, and a past image of the other image acquired before then may bedisplayed.

In the cases of FIG. 26 and FIG. 27, when the user specifies both theframe B and the frame D by a drag operation, the operation signal US inaccordance with the drag operation is transmitted from the userinterface portion 315 to the control portion 310. The operation signalUS includes information of the selected frames B and D. The controlportion 310 transmits the command signal CM, which is generated based onthe operation signal US, to the terminal control portion 210 of thecamera terminal device 200 via the network interface portion 206. Theterminal control portion 210 analyzes the command signal CM, andgenerates the direction control signal CTc based on informationindicating that the two frames B and D are selected and on informationof the preset positions of the frames B and D that is stored in thepanorama image storage portion 207. Thus, the imaging direction of thecamera portion 201 is adjusted, by the imaging direction control portion204 and the imaging direction adjustment portion 205, to a directioninstructed by the direction control signal CTc.

FIG. 28 is a schematic diagram showing, in detail, an example of asetting screen to set the preset position. The preset screen shown inFIG. 28 appears when a “preset position” button shown in FIG. 10 isclicked. In the example of the setting screen shown in FIG. 28, thepanorama image is displayed to the right of the center of the screen,and an image (a live image) of the preset position is displayed to theleft of the center of the screen.

The user can perform a default setting of the preset positions byperforming a drag operation in the panorama image. In this case, forexample, the default setting of each of the frames A to D is performedby dragging, as a start point and an end point, both ends of a diagonalline of the rectangular frame on the panorama image. As a result, theframes A to D that have been set by default are displayed on thepanorama image. After that, in a state where the user has set the framesA to D at desired positions by performing drag operations using themethod shown in FIG. 24 and FIG. 25, when the user sets a preset numberand a name for each of the preset positions and clicks a “Set” button,the preset corresponding to each of the frames A to D is completed.Further, the preset can be called by clicking a “Call” button. Notethat, the settings can be made by inputting coordinates of the presetpositions. In the present embodiment, four preset positions are shown asan example. However, the number of the preset positions is notparticularly limited.

In specifying a “Group preset position” shown in FIG. 28, the presetpositions can be classified into some groups. The groups are expressedas a tree structure. In a “Home setting position” menu, the camera homeposition is displayed.

When a new group folder is created, an attribute of the new group folderis shown in a “New Group” column. A “Rename” button is a button that isclicked when a group name is changed. A “Clear” button is a button thatis clicked when a group is deleted. An “OK” button is a button that isclicked when the settings of this page are stored in the camera. A“Cancel” button is a button that is clicked when a setting change ofthis page is cancelled.

When the user sets the preset position, the operation signal US inaccordance with the user operation is transmitted from the userinterface portion 315 to the control portion 310. The control portion310 transmits the command signal CM, which is generated based on theoperation signal US, to the terminal control portion 210 of the cameraterminal device 200 via the network interface portion 206. The terminalcontrol portion 210 analyzes the command signal CM, and stores thepreset position included in the command signal CM in the panorama imagestorage portion 207. Further, when information of the preset position isstored in the memory provided in the center server 300, the controlportion 310 stores the preset position in the memory based on thecommand signal CM that is generated based on the operation signal US.

As described above, the user can set, change and delete the presetposition(s) using a pointing device, such as a mouse, a keyboard or thelike, in a state where the screen shown in FIG. 28 is displayed on themonitor 320 connected to the center server 300 or on a display portionof the client 400.

Next, setting, on the panorama image, of the position and size of aframe indicating the preset position will be explained. When the userspecifies the position and size of the frame in the panorama image usinga drag operation or the like, the center position of the frame is set asthe preset position. Then, the center position of the frame is stored asthe preset position in the panorama image storage portion 207 or thememory of the center server 300.

A reference size of the frame is set to a size of the frame that ispositioned in the center of the panorama image, because image curvatureoccurs at the periphery of the panorama image. Then, the reference frameis enlarged or reduced to have a size corresponding to an imagingmagnification, and a final size of the frame is determined.

A relationship between the frame size and a zoom amount of the cameraportion 201 of the camera terminal device 200 is set in the followingmanner, for example. As shown in FIG. 28, it is assumed that the size ofthe panorama image is 90 pixels in height and 320 pixels in width. Inthe example shown in FIG. 28, the panorama image is an image (the secondpanorama image) captured by a 360-degree pan operation. Therefore, theangle of view in the horizontal direction corresponds to 360 degrees.

As an example, it is assumed that a horizontal angle of view of a lensof the camera portion 201 is 58 degrees (wide end) and 2 degrees (teleend), and the maximum resolution of the camera portion 210 is 640×480(pixels). In this case, the size (Xw, Yx) (where the unit of Xw and Ywis a pixel (the number of pixels)) of the rectangle when the lens of thecamera portion 201 is set at the wide end is expressed by the followingrelationship.

-   -   360 (degrees) :58 (degrees)=320: Xw        Accordingly, the size of the rectangle in the horizontal        direction is Xw=52 (pixels).

The size Yw of the rectangle in the vertical direction is expressed bythe following relationship.

640:480=52:Yw

Accordingly, the equation Yx=39 (pixels) is obtained. Therefore, thesize of the rectangular frame when the lens of the camera portion 201 isset at the wide end is (52, 39) pixels.

When the camera terminal device 200 is mostly zoomed and the lens is atthe tele end, the size of the rectangle (Xt, Yt) (pixels) is expressedby the following relationship.

360(degrees):2(degrees)=320:Xw

Accordingly, the size of the rectangle in the horizontal direction isXw=1.8 (pixels).

Further, the size Yw of the rectangular frame in the vertical directionis expressed by the following relationship.

640:480=1.7:Yw

Accordingly, the equation Yw=1.4 (pixels) is obtained.Accordingly, as described above, the relationship between the size ofthe frame and the zoom amount of the camera portion 201 is obtained. Thecontrol portion 310 generates the command signal CM based on theposition of the frame specified by the user and on the zoom amount, andtransmits the command signal CM to the terminal control portion 210 ofthe camera terminal device 200 via the network interface portion 206.The terminal control portion 210 analyzes the command signal CM, andcalculates the size of the frame in the preset position based on thezoom amount included in the command signal CM. Then, the terminalcontrol portion 210 stores the calculated size of the frame in thepanorama image storage portion 207 together with the center position ofthe preset position. When the preset position is stored in the memory ofthe center server 300 or the client 400, the control portion 310 setsthe size of the frame based on a zoom position that is included in thecommand signal CM. Further, based on the center position of the framethat is included in the command signal CM, the control portion 310stores the preset position in the memory. Note that, instead ofspecifying the zoom amount based on the size of the frame, the zoomamount may be specified by keyboard input or the like.

As described above, according to the present embodiment, the presetpositions are displayed on the whole image (the panorama image), and itis therefore possible to easily ascertain the preset positions in thewhole image. Further, since the preset can be specified by specifying iton the whole image, it is possible to significantly improve usability.Further, since the size of the preset position (the size of the frame)can be displayed based on an imaging angle of view, it is also possibleto ascertain a relative size of the angle of view that has been preset.Furthermore, even in a case where the whole image and an actuallycaptured image correspond to each other in a distorted manner, the sizeof the angle of view can be ascertained relatively.

Note that, in the above-described explanation, a case is explained inwhich a plurality of users control the imaging direction of each of thecamera terminal devices 200. However, the present invention can also beapplied to a system (a system that performs pan/tilt control digitally)in which the imaging direction of each of the camera terminal devices200 is fixed, and only a region corresponding to a command from a useris cut out from the whole image and displayed. This system is achievedsuch that, for example, the control portion 310 of the center server 300specifies a cut out range from the whole image in accordance with anoperation input of the imaging direction by the user, and performsdisplay control using the display processing portion 305. Also in thiscase, by performing the above-described control using the presetpositions, it is possible to digitally control an imaging range of alive image. As described above, the system of the present embodiment canbe applied not only to the system that mechanically controls the imagingdirection, but also to the system that digitally controls the imagingdirection.

The exemplary embodiment of the present invention is described above indetail with reference to the appended drawings. However, the presentinvention is not limited to the above-described examples. It should beunderstood by those skilled in the art that various modifications,combinations, sub-combinations and alterations may occur depending ondesign requirements and other factors insofar as they are within thescope of the appended claims or the equivalents thereof.

What is claimed is:
 1. A control device comprising: a display controlportion that performs control such that a whole image display region, inwhich a whole image captured by a camera is displayed, and an enlargedimage display region, in which a region corresponding to a positionspecified on the whole image is enlarged and an enlarged image isdisplayed, are displayed; and a control portion that performs controlsuch that the region corresponding to the position specified on thewhole image is changed, wherein the region includes at least one presetregion, and the display control portion displays, on the whole image, asection that corresponds to the preset region.